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audience were directors of small and medium-sized enterprises and those 
responsible for developing ICT use within a company or for managing staff 
training. The project gathered information from previous experience, a 
literature review, and reports; developed the resource on CD-ROM and produced 
a version to pilot; and piloted with business users and evaluated outcomes. 
The resource was organized into 10 interrelated sections. Each section was 
introduced by an audio commentary and had subsections consisting of one or 
more text screens. The sections were introduction, planning, evaluation, 
modes of delivery, six case studies, summary, needs analysis, change in 
practice, glossary, and references. Data for evaluation of the pilot CD-ROM 
were completed questionnaires from 19 business users. Eighteen described 
their use of ICT in a personal context as being well developed or reasonable; 
four users described their business use as well developed. Views were mixed 
regarding navigation around the resource. No consensus was reached as to 
which aspect was most worthwhile. All saw modes of delivery as interesting 
and useful . Issues arising from development and use of the resource related 
to the value of case studies; transferability of good practice; and teamwork. 
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Ensuring Effectiveness of ICT Training: A Report and Evaluation 



Introduction 

Ensuring Effectiveness of ICT Training is a pilot study and resource to inform and 
support the training process. The resource has a target audience of directors of small and 
medium sized enterprises, those with a responsibility for developing the use of 
information and commumcations technology (ICT) within a company, and those 
managing staff traimng. It highlights issues which are important to address if the 
objectives of the ICT development and training are to be achieved. 

Background research was undertaken and the resource developed to this pilot stage by 
the Telematics Centre at the University of Exeter School of Education, working 
collaboratively with the Residential Estate Agency Training and Education Association 
(REATEA). The project was undertaken as an outcome of previous work in this area, in 
particular Training and Assessment: Insuring IT Competence (Davis, 1998). Other 
organisations and companies also contributed to the work, including the University of 
Reading, West Berkshire Educational Business Partnership, Magnetic Fish and a 
number of other companies who were involved in the trial and evaluation of the 
outcomes of the project. The work was undertaken as part of a University of Exeter 
project supported by the European Social Fund. 

This report outlines the context for the report, the research on which it was based, the 
methodology used and an evaluation of a prototype of the resource which was piloted in 
December 1998. The evaluation is based on data collected from trials with 24 users in 
local business, as well as analysis and comment from the development team and 
members of the business community to whom the resource was demonstrated. 

Context 

The need for a better understanding of the training process is widely accepted and has 
been the focus of workshops, seminars and research articles (see, for example, Caley & 
Hendry, 1998; Jones & Robinson, 1998). Whilst some very effective examples of ICT 
training can be cited, such examples may be outweighed by the amount of TT training’ 
where outcomes do not live up to expectations. This situation is not restricted to any one 
sector, probably being equally evident in business, education and industry. 

There is widespread recognition that training is part of any ICT development process, 
and increasing acknowledgement that more than just training and purchase of equipment 
are required if business results are to be achieved from investment in ICT development. 
The education sector is currently addressing this matter through a major initiative TCT 
Training for Teachers and School Librarians’, supported by the National Lottery New 
Opportunities Fund. 

Staff at the Telematics Centre at the University of Exeter have considerable research and 
practical experience in many aspects of ICT training. Over the last few years, they have 
built on this work by developing commercial partnerships in order to understand the 
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needs and practice of the business community, and to research the transferability of the 
education experiences. REATEA, through their role as a national training association, 
have also been looking specifically at the issue of ICT development and the associated 
training needs. Since leading the ‘Technology in Estate Agents’ research and evaluation, 
and playing a lead role in the Insuring IT Competence initiative, they are now producing 
an ‘IT Guide’ with accompanying CD ROM for the industry (REATEA, 1999). 

Ensuring Effectiveness of ICT Training is an outcome of this work. The Telematics 
Centre undertook a siunmary evaluation of the Insuring IT Competence project, which 
was the forenmner to this project (Davis, 1998). This work and the report of IT use in 
the Estate Agency (REATEA, 1997) set a target and standard for this resource. Other 
work which was particularly significant in this project was a series of workshops and 
discussions about models of training with Microsoft, ICL, Granada Learning, Network 
Solutions and Pearson’s Publishing. These were undertaken in the context of developing 
a collaborative model of effective ICT training to schools. As most people involved in 
these workshops were from the business community, it provided a valuable forum 
through which to consider different models of the training process and its component 
parts. The project also ran in parallel to the development of the 'IT Guide', a handbook 
developed by REATEA for Estate Agents (REATEA, 1999). 

Project methodology 

The project methodology consisted of; 

1 . Gathering information from previous experience, a review of the literature (articles 
and reports published on paper and electronically) and from a series of meetings 
with representatives from business and commerce. 

2. Producing a framework and storyboard for the resource through a series of four 
workshops, interspersed with individual writing. 

3. Designing a user interface and identifying images required. 

4. Selecting and representing case studies. 

5. Capturing images, moving video and audio. 

6. Developing the resource on CD ROM and producing a version to pilot. 

7. Piloting with business users and evaluating the outcomes. 

8. Production of project report and dissemination to a wider audience through 
publication. 

Formative evaluation undertaken by the project development team, working closely 
with REATEA, spanned and informed the entire development process. 

Overview of the resource 

The resource is organised into a number of inter-related sections and sub-sections. Each 
section is introduced by an audio commentary and has sub-sections which consist of one 
or more text screens, some with active links to other parts of the resource. Many are 
supported by images and additional sound clips. 
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The Introduction poses a number of questions: What is ICT? Why ICT training? What 
about Health and Safety? What costs? What process? At this stage users are provided 
with a diagram to illustrate the training process visually (see Figure 1). 




Figure 1: The Training Process 



Planning has sub-sections entitled: 

• Training objectives 

• Targeting training 

• Mode of delivery 

• ICT issues 

• Support 

• Preparing for training 

• Planning for change 



Evaluation has sub-sections entitled: 

• Why evaluate? 

• Monitoring change 

• Post training feedback 

• Evaluating the training process 

• Managing the outcomes 

• Identifying changes in practice 



er|c 



Modes of Delivery has sub-sections entitled: 

• Face to Face Training 

• Distance Learning (see Figure 2) 

• Strengths and weaknesses (see figme 3) 

• Models for Training 
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MODES OF DELIVERY 



P Face to Face (F2F) 



p Distance Learning 

Strengths and 
{T wea>messes 



Models for Training 




DtetancB learning can be defined as the uae of any media 
for (Verduin p.124). 

Exampies indude: 



Using a computer 
based training (CBT) 
package, often provided 
on CD-ROM. 



Studying from a 
Teach youraetf 
book or manoal. 






Learning on-Qne 
ie over the 
Internet 



P Case Studies -| 



Qloduory rvlerence* 



Suppoitod and unsupported distance learning 

An important discnminaling issue is to what extant and in wtiat 
way. Vio training is SM^ported. io how is hob accessed, what 
oppoilmity is there lor discussion and iiSerBCtivB activity, is there 
anyone In ate role of tutor or manager of learning? 

i ; next ^ 



Figure 2: Modes of Delivery: Distance Learning 




Figure 3: Strengths and weaknesses: provided on the CD ROM as an interactive grid. 

There are six Case Studies: BEST COPY AVAILABLE 



• Applying communication technology in SMEs (actS); a WWW based course for 
the business community in the use of communications technology. 

• Support action to facilitate the use of video conferencing in education (saVie); a 
course in video conferencing for businesses. 

• South Bristol Learning Network (SBLN); a case study showing how a small 
business, the Badminton Villa Hotel, developed ICT skills and applied them to 
their own context through a training opportunity with SBLN. 
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• UpGrade media and training services (UpGrade); an example of a bespoke 
course provided to Devon and Cornwall Careers by UpGrade, a University- 
based business. 

• The Virtual Study Centre (VSC); an example of an on-line environment set up 
and resourced for collaborative learning with tutor support. 

• RATIO (Rural Area Training and Information Opportunities) is an example of a 
how anyone can use the facilities of a 'drop in' centre to follow an accredited IT 
course. 

For each there is an overview, key features, a training description and a showcase. 

The Summary contains sub-sections entitled: 

• attitude and commitment 

• integration into the workplace 

• training structure and commitment 

• business plan 

• evaluation 

The Needs Analysis section has summary remarks and a list of characteristics of a good 
needs analysis. 

The Change in Practice section contains summary remarks. 

The Glossary is limited in breadth at this stage as it illustrates an approach; it 
incorporates some audio linked to sequences of still images. 

The References section provides WWW site addresses and references to articles and 
reports. 

The pilot study 

The evaluation of the pilot CD-ROM was informed by three sources of data: 

1 . Feedback from questionnaires completed by business users who piloted the resource 
in their workplace. 

2. Comments collected during demonstration to members of the local business 
community. 

3. Reports from the development team. 

The responses of the first group form the basis of the analysis in this section. The 
second and third of these sources are viewed as formative evaluation which was used 
mainly to inform the development of the pilot CD-ROM. 

Copies of the resource were given to 24 people from local business, together with an 
overview sheet and questionnaire (see Appendix). The questionnaire asked users about: 

• the level of their personal use of ICT 

• the level of their use of ICT in business 

ERIC 
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• the ease of use of the resource 

• the usefiilness of sections in the resource 

• the most worthwhile and least interesting parts of the resource 

• suggestions for changes or developments 

The Telematics Centre and REATEA were both involved in setting up and conducting 
the pilot study which comprised 12 people from within the estate agency profession and 
12 from other businesses. The resource was sent initially to the ‘mixed business’ group 
and later to the estate agents. The two-staged approach was to ensure the trial was 
undertaken at a time when people would be able to give some priority to using the 
materials. For the group of Estate Agents, this was a one month period leading up to 
their quarterly steering group meeting which was extended to incorporate a ‘post-trial 
workshop’ to discuss the issues and collect additional comment. 

Outcomes 

Nineteen users returned completed questioimaires. Of these, all but one described their 
use of ICT in a personal context as being well developed or reasonable. The use of ICT 
in the business was generally lower. There were only 4 users who described their 
business use as well developed. In only one case did someone report the ICT use in their 
business as being better developed than their own personal use. It was interesting that 
this one case was that of a user who described themselves as having quite limited 
personal use of ICT, but felt the business use was well developed. Others in the same 
office described the ICT use in the business as quite limited. 

Twelve of the users testing the resource with no previous viewing or support, said that 
the resource was easy to use, with the remaining 5 who responded to that question 
saying mostly OK. The interface was described as being very “ clean” with generally 
good-quality images, sound and text. Some users reported technical difficulties in 
getting the sound and video to play. 

The navigation around the resource prompted comment and discussion. Views were 
mixed as to whether there should have been a clearer and more structured route or 
additional links between sections. One user noted that only some pages had a previous 
arrow. This was felt to be helpful, and it was suggested that a next arrow could also be 
offered for those who preferred to follow a pre-defined route. 

Various sections caught the interest of users in response to being asked for the aspect of 
the resource they found most worthwhile, but there was no real consensus of views. 
Several users said that they particularly liked the visual style of the health and safety 
sub-section, and all but three respondents referred to this section as interesting and 
use ful. There was keen interest in the health and safety section in the post trial workshop 
to the extent that the project team were invited to develop this section further for 
inclusion on the CD ROM being developed as part of the ‘IT Guide for Estate Agents’. 

Modes of delivery was seen by all as interesting and useful with the strengths and 
weaknesses grid singled out in several cases for positive comment. It became evident in 

_ ll 
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post-trial discussion that not all users had appreciated the interactive nature of the grid, 
and it was noted that this needed to be made clearer. 

Case studies were popular, and it was suggested that the case studies were most useful 
when they were written from the client's angle, as the South Bristol Learning Network 
(SBLN) did through the proprietors of the Badminton Villa Hotel. There was a call for 
more case studies or references to existing good practice. 

The question regarding the section of least interest was not addressed by all 
respondents. Of those who did respond. Measuring Change was noted by two users and 
planning and evaluation were each noted by one user. 

The importance of needs analysis in the training process was referenced by two people 
who felt this section could be developed further, possibly with some structures included 
for audit tools etc. 

In general the sections which provided a ‘multimedia’ approach, i.e. incorporated 
images and sound, were preferred to those with a text-based approach. Several people 
commented on this aspect in discussion or through the questionnaire and asked for more 
use of video. 

The final question invited comment regarding suggested changes or future development. 
A summary of the responses is given in the table below. 




Concentrate on the target audience 
Expand the glossary 
Increase the use of multimedia 

Perhaps more videos as these always maintain attention span 

Main menu could be flagged up more initially 

Some topics had a 'previous' arrow. This would be handy throughout 

Instructions on moving between screens not always easy to find/follow 

What about a 'print this page' option? 

Summary doesn't shout out enough 

You could have tiny samples of case studies in text and sound flags to give a hands on descriptive feel for those 
who still need to be persuaded, or to whet the appetite 

On the first day I had sound and then had an error message 

Technical problems like video and sound 

Opportunity to hyperlink to http://www.ratio.org.uk etc 

Face to Face and Distance learning need separate screens 

Needs analysis to give indication of competencies/ICT skills relative to job profile 

More case studies referenced to good practice 

It would be useful to have links or references to relevant courses for trainers and relevant ICT courses for trainees 
Have more of it interactive 
Concentrate on the target audience 
Provide updating service 



Table 1: Responses from 'Suggested changes or developments' 
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Issues arising from the development and use of the resource 

In earlier developments at the Telematics Centre, the importance of context or the 
situational dependency of multimedia has been acknowledged. Users of multimedia 
resources bring with them unique experiences and have different expectations. Although 
these variables can never be wholly accommodated, multimedia designed for 
educational applications can be targeted at specific courses and structured to meet pre- 
determined, collectively agreed objectives (Dillon et al, 1999; Tearle et al, 1999). 

The Ensuring Effectiveness of ICT Training resource was designed to support 
occupational learning; that is, learning which takes place at work. The workplace is a 
complex arena and although it is an educational setting in the broadest sense, in many 
fundamental respects it is different from formal educational environments and presents 
different challenges for the design of multimedia. 

The workplace is characterised by a number of different forms of knowledge; Blackler 
(1995) has suggested the following (see also Saunders, 1998): 

• embrained knowledge, which is dependent on conceptual skills and cognitive 
abilities 

• embodied knowledge, which is action-orientated and likely to be only partly 
explicit. It is knowledge of “how we do things here” 

• encultured knowledge, which refers to the process of achieving shared 
understandings; it is language-based, socially constructed and negotiable 

• embedded knowledge, which resides in systemic routines, roles and formal 
procedures 

• encoded knowledge, which is information conveyed by signs and symbols 

These forms of knowledge usually have an especially high level of situational 
dependency. There is tremendous variation from company to company in how things are 
done and in formalised and encoded routines and procedures. Each company has it own 
‘culture’, that is, shared knowledge, beliefs, ways of acting and norms governing social 
interaction (Saunders, 1995). Not only may the culture of one company be very different 
from that of another operating in the same sector, it may also have a commercial 
sensitivity and be subject to levels of confidentiality absent from the formal educational 
environment. 

For these reasons, multimedia which is designed to have widespread and open 
applications in occupational learning has to be based on generic rather than specific 
content and to focus primarily on processes. There is a paucity of such resources (Davis, 
1998); companies which have very specific operational training needs normally 
commission bespoke materials. The Ensuring Effectiveness of ICT Training resource 
thus focuses on the generic elements of ICT training. It has at its core a process model 
which starts with needs which inform planning and decisions about mode of delivery, 
which in turn determine the form of training and the method of evaluation, particularly 
of managing outcomes and monitoring change in practice. The outcomes of the 
evaluation feed back into the analysis of needs. These processes are common to most 
commercial training situations. 
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Individuals from different backgrounds bring with them different assumptions about 
what constitutes ‘quality’ and often unconsciously apply these when making judgements 
about current activities (Scott, 1996). During the pilot there was no consensus about the 
most and least useful sections of the resource, but the importance of a needs analysis in 
the training process was stressed and the section on Modes of Delivery was seen by all 
as interesting and useful. The lack of a consensus view about the most and least useful 
sections is partly a reflection of the diversity of business cultures explained above. 
However, it also reflects a similar situation in higher education where people make 
individual and often highly specific choices about how they manage information 
(Dillon, 1998). 

Typically, there is a difference in the value placed on case studies by business and 
educational users. This is partly a matter of context and personal relevance (see above) 
and partly a reflection of real cultural differences between educational and corporate 
training environments and their goals (Weller & Dillon, 1999). One business user in the 
pilot viewed the Virtual Study Centre, although not used exclusively by education, to be 
‘educational’ and thus wondered if it was relevant to business users. This was included 
after some debate in order to provide an example of an on-line learning environment. It 
was felt that many users of the resource would be unfamiliar with such a concept. It is 
therefore relevant to note that whereas the values of commerce and education are 
profoundly different and incommensurable (Tasker & Packham, 1993), some of the 
theories of learning which have been influential in education and multimedia 
development in recent years, including Kolb’s theory of experiential learning (Sadler- 
Smith, 1996), Schon’s concept of reflective practice (Thorpe, 1995), the notion of 
mastery learning (Palardy, 1993), the notion of situated cognition (Brown et al, 1989; 
Lave, 1996) and Pask’s conversational theory (Pask, 1976; Laurillard, 1995), have their 
origins at least in part in commercial settings. The Ensuring Effectiveness of ICT 
Training resource uses case studies drawn from business and educational settings, 
including education providing businesses with training solutions, and focuses on the 
common concerns by drawing on examples of best practice from both sectors. 

Good practice in ICT is transferable from one sector to another but only if context is not 
allowed to get in the way. However, in the absence of measurable outcomes which 
relate directly to the commercial interests of a company, ICT training is rarely valued 
(Davis, 1998). This is a common problem; Boucher (1998) noticed a similar reluctance 
on the part of senior managers in higher education institutions to recognise the value of 
transferable skills in ICT if they cannot be equated with efficiency gains in 
administration and teaching or research output. (Skills in this sense carries a positive 
connotation of competence, of practical knowledge combined with ability [Eraut, 

1994]). 

Teamwork is another important non-quantifiable benefit arising from collaborative 
multimedia development (Dillon et al, 1998), in this case between personnel in the 
Telematics Centre working in partnership with members of the local business 
community. Teamwork is good for the development of products, and the process of 
collaboration results in shared skills and expertise and provides fertile ground for the 

^ ilt 
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emergence of new ideas and perspectives. One key outcome of the project was the 
collaboration between education and business. Two of these links are likely to lead to 
further collaborative ventures. Campbell et al (1995) note that inter-institutional 
collaborative partnerships are, by definition, alternative approaches to solving problems 
or improving practice, and as such are innovative in their own right. They also note that 
collaborative partnerships quite often ‘evolve’ in response to sudden opportunities that 
are available for limited periods of time. 

Users have high expectations of multimedia; they judge audio-visual material by 
broadcast standards and interactivity by the standards of commercial software. They 
have low tolerance of material which falls short of these expectations. A strength of 
Ensuring Effectiveness of ICT Training is the way it uses multimedia techniques 
developed from earlier projects within the Telematics Centre. The value does not lie 
merely in exemplifying the use of ICT in education and training, but also in showing 
processes which are common to both. 

Recommendations for further development 

The evaluation suggests that the pilot resource is functional and well structured but that 
it could be extended and made more interesting. The recommendations that follow are 
thus concerned mainly with enhancing interactivity and further developing those 
sections which are seen by the business community as particularly useful. 

In the course of developing, trialling and evaluating earlier resources, staff at the 
Telematics Centre have developed extensive expertise in the use of images, both stills 
and videoclips, to enhance and extend education and training. The strategies for using 
images have proved to be successful in parts of the pilot version where this approach 
has been used. It is planned to extend this across the resource; for example, more images 
will be included. Some of these will be made active to provide further information 
(which may also be image-based) about the content or context being depicted as has 
been done for the health and safety sub-section. Similarly, the sub-section on the 
analysis of strengths and weaknesses presents an active grid. At the moment each part of 
the grid activates a text screen. The development team would also like to include visual 
exemplification or an embedded activity or task. 

Interviews were undertaken with a number of people following ICT training, as well as 
with training providers. Extracts from those interviews, offered as sound bytes, will also 
be added to the resource to illustrate points made in the text. 

Given the importance of needs analysis (a view supported by most users of this 
resource), this section needs to provide a much more comprehensive coverage of the 
area. A diagram could be included with the list of characteristics of a good needs 
^alysis and perhaps some sample questionnaires and audit tools. One pilot user 
suggested including a means of showing competence in ICT skills relative to job 
profiles. Another suggested a self-diagnostic tool and/or a task to help users determine 
their own learning preferences and styles. 
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The visual glossary demonstrates an approach to the explanation and clarification of 
terms. The pilot sought to discover people’s comments on this approach in order to 
inform further development and breadth of coverage. As the approach was generally 
valued by the business users, there is scope for developing it further and for 
incorporating some interactivity and cross-referencing. 

The case studies were seen as useful, and it is felt some should be reviewed in order to 
show more of the trainee's view of the experience, and the impact the training had on 
their business. It was also felt the examples should be made more interactive in order to 
show them as they operate in their own right. For example: 

• The plan of the Virtual Study Centre in everyday use is a ‘clickable image’, used 
to navigate the environment. More of this realism needs to be made available in 
the Ensuring Effectiveness of ICT Training resource to allow active site access to 
each of the rooms (rather than a next screen sequence). This could be achieved 
through an activity or task embedded in each so that the user can get an idea of 
how the Virtual Study Centre actually works. 

• acts shows ‘test yourself and ‘self analysis’ facilities which are active in the 
real course. It was felt they should be made active in this case study. 

It was suggested that navigation could be enhanced by adding 'previous' and 'next' 
arrows. More cross-referencing links also needed to be considered, though it was 
appreciated that any additional cross-referencing must be carefully thought through or it 
could result in users feeling ‘lost’ in the resource. A ‘ resource map’ icon is now under 
consideration to address this issue. 

There is always a problem of updating a CD-ROM resource and some business users in 
the pilot study asked for live links to relevant web sites. Some such sites are given in the 
references, but at present do not operate as live links. Updates and additional material 
can also be offered on-line. Both of these matters could be resolved if a version of the 
resource were offered on-line. In this case a ‘print the page’ facility would offer further 
flexibility. 

Finally, as was noted above, the benefits of developing transferable skills are difficult to 
track through conventional models of cost-benefit analysis. At the same time, 
government initiatives are encouraging the development of transferable skills. These 
initiatives include the University for Industry which will involve the creation of a 
national multimedia learning network, the Investors in People Standard which links 
investment in training to the needs of businesses, and skills accreditation through 
Training and Enterprise Councils and the National Vocational Qualifications 
Framework. These proposed networks, as well as existing ones, could be used as a 
means of disseminating these materials to a wider audience. Accreditation could also 
provide a major incentive for using the resource. 

Conclusion 

Ensuring Effectiveness of ICT Training is a functional resource developed in an area 
where there is a paucity of good-quality multimedia, i.e. it addresses the training 
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process and does not attempt to carry out any ICT skills training. It has been piloted in 
a limited number of business settings and its value has been acknowledged. It could be 
improved with further development of its interactive and multimedia elements and by 
extending the content signalled by the users as being particularly useful. 

The process of developing the resource has resulted in many new links being established 
several of which look set to lead to further collaboration beyond the life of the project. 
The work has provoked discussion and reflection within and between business and 
education communities and undoubtedly has been a valuable learning experience and 
training activity in its own right. 
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